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ABSTRACT 


The  Aviation  Selection  Test  Battery  (ASTB)  has  been 
used  to  screen  prospective  aviation  candidates  for  the  Navy, 
Marines,  and  Coast  Guard  since  World  War  II.  The  Navy  has 
continuously  looked  for  ways  to  update  and  improve  its 
aviation  screening  procedures  and  the  latest  supplement  of 
the  Performance-Based  Measurement  Battery  (PBMB)  currently 
under  development  is  just  one  example.  The  Naval  Operational 
Medicine  Institute  (NOMI)  is  currently  investigating  how 
well  this  test  will  assist  with  the  difficult  process  of 
aviation  candidate  selection.  This  study  was  conducted  to 
assess  whether  individuals  with  aviation  experience  would 
perform  better  on  the  PBMB  than  those  with  no  aviation 
experience.  Forty  individual  participated  in  this  research, 
20  had  formal  aviation  training.  The  results  showed  that 
experienced  aviators  performed  significantly  better  on  eye- 
hand  coordination  tracking  tasks  than  the  group  with  no 
aviation  experience. 
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EXECUTIVE  SUMMARY 


This  thesis  performed  an  assessment  of  the  Navy' s 
Performance-Based  Measurement  Battery  (PBMB)  under 
development  and  intended  to  supplement  the  Aviation 
Selection  Test  Battery  (ASTB) .  These  findings  will  augment 
the  validation  study  currently  being  conducted  by  the  Naval 
Operational  Medicine  Institute  (NOMI)  in  Pensacola,  Florida. 

A  total  of  40  individuals  attending  the  Naval 
Postgraduate  School  volunteered  to  participate  in  this 
study.  Of  those  40  volunteers,  20  had  received  formal 
aviation  training  as  either  a  pilot  or  Naval  Flight  Officer 
(NFO)  .  All  participants  were  administered  the  test  battery 
which  consisted  of  three  components:  a  direction  orientation 
test,  a  dichotic  listening  test  (DLT) ,  and  a  multi-tracking 
task.  Combinations  of  these  components  were  then  used  to 
make  up  two  multi-tasking  tests:  the  multi-tracking  with  DLT 
subtests  and  the  multi-tracking  with  the  emergency 
procedures  (EP)  subtests. 

Results  indicate  that  those  with  aviation  experience 
performed  better  on  all  multi-tracking  tasks.  The  aviation 
experience  group  and  the  group  without  aviation  experience 
showed  the  largest  significant  difference  on  the  multi¬ 
tasking  test  comprised  of  the  multi-tracking  and  emergency 
procedures  (F  =  12.49,  p  =  .001)  .  Background  demographics 
like  the  amount  of  video  games  played  were  determined  to  be 
immaterial.  The  aviation  experience  group  and  the  group 
without  aviation  experience  did  not  differ  in  regard  to  the 
performance  on  the  direction  orientation  test  (F  =  0.57,  p  = 
.48)  . 

xiii 


Overall,  the  PBMB  was  capable  of  detecting  important 
eye-hand  coordinated  tracking  skills  and  with  further 
analysis  and  refinement  to  the  scoring  algorithm,  this  test 
battery  should  improve  future  aviation  candidate  selection. 
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I. 


INTRODUCTION 


A.  BACKGROUND 

Since  the  armed  forces  of  the  United  States  started 
using  aircraft  in  combat,  there  has  been  a  dedicated  effort 
to  make  certain  that  the  right  candidates  are  selected  for  a 
career  in  military  aviation.  The  United  States  Navy  (USN) 
quickly  recognized  that  the  individuals  selected  needed  to 
possess  the  appropriate  physical  and  mental  attributes 
required  to  become  successful  aviators.  Flying  an  aircraft 
has  always  been  considered  a  dangerous  and  costly 
occupation.  The  costs  required  to  properly  train  and  equip 
a  pilot  or  flight  officer  have  continued  to  escalate.  It  is 
important  that  an  organization  have  the  ability  to 
accurately  select  the  right  individual  from  a  pool  of 
applicants.  In  1993,  training  a  single  Student  Naval 
Aviator  (SNA)  in  a  jet  aircraft  exceeded  $900,000  (Hewes, 
1994) .  Given  that  this  is  such  an  expensive  evolution,  the 
USN  makes  every  attempt  to  minimize  the  chance  of  selecting 
an  individual  without  the  proper  skill  sets  into  the  initial 
aviation  candidate  selection  process. 

This  monetary  cost  drives  the  Navy,  Marine  Corps,  and 
Coast  Guard  services  to  keep  flight  school  failure  rates  as 
low  as  possible.  Improvement  efforts  resulted  in  the 
development  of  the  Aviation  Selection  Test  Battery  (ASTB) . 
Since  the  adoption  of  the  ASTB,  it  has  been  consistently 
used  as  the  criterion  for  selecting  officer  aviation  program 
applicants.  The  ASTB  was  revised  by  the  Naval  Operational 
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Medicine  Institute  (NOMI)  in  cooperation  with  the 
Educational  Testing  Services  (ETS)  in  Princeton,  New  Jersey, 
in  1992  (NOMI,  28  Sep  2007)  . 

The  United  States,  particularly  NOMI,  has  continued  to 
make  improvements  in  the  ASTB  since  its  initial  creation. 
Not  all  the  proposed  modifications  have  been  formally 
adopted  but  those  that  made  the  cut  have  proven  to  improve 
the  selection  process.  Many  of  these  tests  were  created 
during  the  World  War  II  era  and  remain,  for  the  most  part, 
unchanged  (Griffin,  1996) .  A  prime  example  is  the 
perceptual-cognitive  "paper  and  pencil"  type  examinations. 

The  ASTB  has  been  a  successful  aptitude  test  in  the 
candidate  selection  process.  This  test  battery  assesses  an 
applicant's  math  skills,  the  ability  to  extract  meaning  from 
written  material,  familiarity  with  mechanical  concepts  and 
simple  machines,  and  the  ability  to  perform  mental  rotations 
in  order  to  determine  the  spatial  orientation  of  aircraft  in 
3-dimensional  space.  The  ASTB  also  measures  an  individual's 
knowledge  of  aviation  and  nautical  terminology,  familiarity 
with  aircraft  components  and  function,  knowledge  of  basic 
aerodynamic  principles,  and  the  ability  to  grasp  flight 
rules  and  regulations.  From  the  entire  test  battery,  four 
scores  are  derived  from  the  combinations  of  the  subtests. 
The  predictive  validity  of  the  Academic  Qualifications 
Rating  (AQR)  (used  in  prediction  of  SNA  academic  grades)  is 
r  =  0.45  (p  <  .001),  while  the  validity  of  the  Pilot  Flight 
Aptitude  Rating  (PFAR)  (used  to  predict  SNA  flight  grades) 
is  r  =  0.35  (p  <  .001) .  The  current  test  battery  is  viewed 
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by  many  experts  as  adequate  in  its  ability  to  predict 
individuals  who  will  succeed  during  aviation  training  (NOMI, 
28  Sep  2007 )  . 

Although  the  current  test  battery  does  have  a  proven 
track  record,  it  could  benefit  from  further  improvement.  One 
particular  area  under  refinement  is  psychomotor  testing.  An 
appropriate  eye-hand  coordination  test  would  be  beneficial 
in  refining  the  ability  of  the  ASTB  to  predict  whether  or 
not  a  Student  Naval  Aviator  (SNA)  will  succeed  during  the 
flight  portion  of  aviation  training.  According  to  Damos 
(1996),  the  vast  majority  of  current  pilot  selection 
batteries  are  better  predictors  of  training  performance 
rather  than  operational  performance.  Others  share  a  similar 
belief  on  the  present  aviation  test  batteries. 
Additionally,  Damos  along  with  McFarland  (1953)  believe  that 
data  obtained  from  observational  methods  do  not 
appropriately  reflect  some  of  the  more  important  aspects  of 
a  pilot's  job  which  are  the  cognitive  and  psychomotor  skill 
sets  (Damos,  1996) .  A  task  analysis  performed  by  NAVAIR  has 
shown  that  eye-hand  coordination  is  one  of  the  most 
important  physical  skills  candidates  should  possess  (Mangos, 
2005)  . 

The  Federal  Aviation  Administration  (FAA)  also  believes 
that  psychomotor  or  physical  skills  are  important  traits  for 
aviators.  These  types  of  skills  are  required  to  conduct  many 
routine  tasks  when  flying  an  airplane.  Besides  basic 
airmanship,  some  typical  activities  which  involve  eye-hand 
coordination  include  the  ability  to  fly  a  precision 
instrument  approach  procedure,  programming  a  GPS  receiver, 
or  using  sophisticated  maintenance  equipment.  As  physical 
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tasks  and  equipment  become  more  complex,  the  requirement  for 
inteqration  of  cognitive  and  physical  skills  increases  at  a 
proportional  rate  (FAA,  1999)  . 

B.  PSYCHOMOTOR  TESTING 

The  idea  of  using  eye-hand  coordination  testing  in 
aviation  selection  is  not  a  new  concept.  As  early  as  the 
1930's,  tests  involving  psychomotor  skills  proved  to  be 
reasonably  successful.  During  World  War  II,  the  primary 
psychomotor  test  used  for  pilot  selection  was  the  Mashburn 
automatic  serial-action  complex  coordinator.  This  test 
involved  a  timed  coordination  of  stick  and  rudder  movements 
in  response  to  40  different  visual  patterns  and  proved  to  be 
a  good  predictor  of  performance  (DeHart,  2002) .  This 
apparatus  did  have  a  down  side.  It  was  bulky  and  special 
training  was  required  to  assemble  and  collect  data.  Because 
logistical  requirements  made  it  difficult  to  transport 
between  testing  sites,  its  use  was  discontinued  by  1951 
(Deckart,  1988) . 

Recent  improvements  in  modern  technology  have  allowed 

the  Navy  to  directly  address  some  of  these  limitations.  The 

cost  incurred  in  managing  hardware,  software,  and  technical 

support  for  such  a  large  quantity  of  testing  sites  has 

become  less  expensive  and  more  manageable.  Additionally, 

the  availability  of  the  personal  computer  and  off-the-shelf 

hardware  such  as  joysticks,  throttles,  and  rudder  pedals  has 

allowed  psychomotor  testing  to  become  less  cumbersome  and 

easier  to  administer  to  applicants.  Internet  connectivity 

has  provided  an  expanding  reach  in  testing,  scoring,  and  the 

maintenance  of  an  accurate  database  on  all  prospective 

aviation  candidates.  These  advancements  have  allowed  the 

4 


aviation  community  to  expand  eye-hand  coordination  testing 
options  and  develop  relevant  tests  to  determine  which 
particular  tasks  can  help  refine  the  aviation  selection 
process.  These  advancements  have  resulted  in  approximately 
150  Navy  testing  facilities  across  the  United  States  and  on 
average  10,000  Navy,  Marine,  and  Coast  Guard  applicants  a 
year  taking  the  ASTB  (Olde  and  Walker,  2006) . 

Since  these  tests  have  become  easier  to  develop  and 
administer,  studies  have  been  conducted  as  to  their 
relevance.  Delaney  (1992)  evaluated  the  use  of  a 
psychomotor  task  (PMT)  in  the  aviation  selection  process. 
Included  with  this  particular  PMT  was  a  dichotic  listening 
task  (DLT) .  The  statistical  evaluation  of  the  automated  DLT 
and  PMT  concluded  that  both  skills  contributed  to  the 
prediction  of  primary  flight-training  criteria  (Delaney, 
1996) .  The  results  from  this  study  suggested  that  assessing 
concurrent  psychomotor  tracking  and  DLT  performance  may  be  a 
particularly  effective  way  of  predicting  performance  in 
flight  training.  Large-scale  validation  studies  of  the  Air 
Force's  computerized  psychomotor  test  have  reinforced  these 
findings  (Carretta,  1989). 

Delaney  (1992)  determined  that  psycho-motor  based 
measures  could  account  for  an  additional  14.8%  of  unique 
variance,  above  what  was  already  being  accounting  for  by  the 
ASTB.  Griffin  and  Koonce  (1996)  had  similar  results  and 
concluded  that  psychomotor  based  tests  could  explain  16% 
more  variance  than  the  standard  paper  and  pencil  version  of 
the  ASTB.  These  two  studies  indicate  that  psychomotor  tests 
can  explain  approximately  15%  more  unique  variability 
between  aviators  who  get  low  flight  grades  versus  those 
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aviators  who  get  high  flight  grades.  This  may  be 
contributed  to  the  fact  that  a  psychomotor  test  measures 
eye-hand  coordination  and  tracking  skills,  skills  not 
currently  assessed  by  the  standard  cognitive  based  version 
of  the  ASTB  (Olde  &  Walker,  2006)  . 

The  Navy's  Performance-Based  Measurement  Battery  (PBMB) 
is  a  group  of  timed,  interactive  psycho-motor  subtests 
assessing  a  number  of  different  skills  and  abilities 
including  multi-tasking,  eye-hand  coordinated  tracking,  task 
prioritization,  decision  making,  and  spatial  orientation. 
These  skills  and  abilities  have  been  shown  to  be 
particularly  important  in  aviation  (FAA,  1999).  Detailed 
directions  are  displayed  on  the  computer  screen  prior  to  the 
commencement  of  all  seven  subtests.  The  PBMB  also  provides  a 
practice  session  for  all  the  subtests  except  the  last  one 
(an  emergency  procedures  test) .  Also,  all  subtests  had  to  be 
completed  within  the  allocated  time  otherwise  the 
participant  would  not  receive  a  valid  score. 

This  thesis  is  intended  to  assess  the  performance  of 
the  current  version  of  the  PBMB,  which  is  still  under 
development  and  intended  for  use  to  supplement  the  Navy' s 
ASTB.  These  findings  will  augment  the  validation  study 
being  conducted  by  NOMI  in  Pensacola,  Florida.  Specific 
questions  addressed  in  this  thesis  include: 

Do  individuals  with  aviation  experience  perform  better 
on  the  PBMB  than  those  with  no  flight  experience? 

Do  individuals  who  play  sports,  video  games,  or  have 
other  unique  demographic  background  characteristics  perform 
better  on  the  PBMB? 
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Do  participants  have  any  usability  issues  when  taking 
the  PBMB? 

This  project  is  in  support  of  the  Navy's  effort  to 
develop  a  psychomotor  test  that  can  successfully  identify 
applicants  with  good  eye-hand  coordination  and  multi-tasking 
abilities  prior  to  being  accepted  into  the  aviation  training 
pipeline . 
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II.  METHODS 


The  latest  version  of  the  PBMB  was  administered  to  40 
volunteers.  This  sample  consisted  of  two  subgroups.  The 
first  group  was  comprised  of  volunteer  test  subjects  with 
formal  aviation  experience  while  the  other  group  had  no 
aviation  experience.  We  hypothesize  that  the  qualified 
aviators  on  average,  possess  more  fine  tuned  psychomotor 
skills.  We  expected  this  group  to  average  a  higher  score  on 
the  psychomotor  test  than  the  group  with  no  aviation 
experience.  It  was  beyond  the  scope  of  this  study  to 
investigate  how  the  aviation  group  developed  better  eye-hand 
coordination  skills.  We  simply  assumed  that  they  possessed 
those  skills  and  that  the  battery  would  accurately  assess 
this  difference.  We  realize  that  these  traits  could  be  a 
manifestation  of  inherent  natural  ability  or  learned  through 
countless  hours  of  formal  aviation  training  and  experience. 
If  humans  have  eye-hand  coordination  as  an  inherent  trait, 
then  it  is  anticipated  that  some  of  the  participants  with  no 
formal  flight  training  would  score  as  well  as  those 
individuals  with  aviation  experience,  but  as  a  group  those 
with  no  formal  flight  training  should  produce  a  lower 
average  score.  The  primary  goal  of  this  research  is  to 
explore  the  fidelity  of  this  new  selection  test  battery  and 
investigate  whether  or  not  this  particular  version  of  the 
PBMB  can  correctly  identify  important  characteristics  such 
as  eye-hand  coordination  and  spatial  orientation  ability. 
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A.  SUBJECTS 

The  PBMB  was  administered  to  40  volunteers  attending 
the  Naval  Postgraduate  School  located  in  Monterey  California 
during  a  three  week  period  in  the  Fall  of  2007.  All  but  one 
of  the  volunteers  was  serving  on  active  duty  and  nine 
individuals  were  representatives  from  five  different  foreign 
military  services. 

B .  APPARATUS 

Advancements  in  computers  and  software  technology  have 
resulted  in  a  substantial  improvement  in  the  way  selection 
tests  can  be  administered.  They  have  also  allowed  for 
improvements  in  the  level  of  sophistication  and  realism 
contained  in  current  psychomotor  tests.  The  version  used 
during  this  research  required  applicants  to  use  a  joystick, 
throttle,  and  headphones  along  with  other  standard  computer 
hardware  to  complete  the  test  battery.  All  the  hardware 
used  is  available  for  purchase  off  the  shelf  and  the  APEX 
delivery  system  allows  for  the  test  to  be  administered 
online . 

Participants  were  positioned  in  a  quiet,  comfortable 
environment  at  standard  office  desk  and  chair.  The  layout 
used  for  this  experiment  is  depicted  in  Figure  1.  The 
computer  keyboard  was  required  to  complete  the  demographic 
section.  Once  the  demographic  information  was  filled  out, 
the  mouse  and  headphones  were  used  to  perform  the  first 
subtest,  direction  orientation  test  (DOT) .  Upon  completion 
of  the  DOT,  the  keyboard  was  set  aside  and  a  joystick  and 
throttle  were  setup.  The  joystick,  throttle,  and  headphones 
were  used  for  the  remainder  of  the  examination. 
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Figure  1.  Sitting  environment  used  during  the  PBMB . 

This  research  project  used  a  Thrustmaster  HOTAS  Cougar 
combination  throttle  and  joystick,  shown  in  Figure  2.  These 
controls  were  designed  to  replicate  those  used  in  an  F-16 
jet  aircraft.  For  the  purpose  of  standardization,  the 
factory  settings  on  both  the  joystick  and  throttle  were 
maintained  throughout  the  experiment,  however  participants 
could  maneuver  them  into  a  comfortable  position  as  long  as 
the  throttle  was  positioned  on  the  left  and  the  joystick  was 
either  centered  or  on  the  right. 
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Figure  2 .  Thrustmaster  HOTAS  Cougar  throttle  and 
joystick  combination  used  for  the  PBMB . 

C .  PROCEDURES 

The  PBMB  consisted  of  the  following  subtests  in  the 
following  order: 

•  Direction  Orientation  Test  (DOT) 

•  Dichotic  Listening  Test  (DLT) 

•  Vertical  Tracking  Test  (VTT) 

•  Airplane  Tracking  Test  (ATT) 

•  Vertical  Tracking  Test/Airplane  Tracking  Test 

(VTT/ATT ) 

•  VTT/ATT/DLT  Multi-task  Combination 

•  VTT/ATT/Emergency  Procedures  (EP) Scenario 

Besides  the  direction  orientation  test,  the  PBMB  was 
designed  for  each  subsequent  section  to  build  on  each  other 
and  provide  practice  with  these  novel  tasks.  The  final 
composite  score  will  be  comprised  of  the  DOT,  the 
VTT/ATT/DLT,  and  the  VTT/ATT/EP  subtests. 
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The  ASTB  PBMB  contained  a  demographic  section  which  was 
required  to  be  completed  before  starting  the  examination  but 
not  used  in  this  research.  Questions  included  with  the 
software  were  name,  gender,  race,  place  of  birth,  date  of 
birth.  Social  Security  Number  (SSN) ,  present  military 
status,  prior  military  service,  educational  level.  Grade 
Point  Average  (GPA)  for  highest  completed  education  level, 
major,  experience  with  flight  simulator  games  or  software, 
any  formal  flight  training,  any  aviation  training  -  what 
level  and  number  of  hours. 

For  the  research  conducted  and  contained  in  this 
thesis,  all  the  participants  were  asked  a  different  set  of 
background  and  demographic  questions  designed  specifically 
to  gauge  a  participant's  eye-hand  coordination. 
Additionally,  any  personal  information  such  as  a  SSN  was 
substituted  with  a  subject  number  to  assure  anonymity.  An 
example  of  the  questionnaire  is  located  in  Appendix  A. 

1.  Direction  Orientation  Test  (DOT) 

The  direction  orientation  test  is  the  first  subtest  of 
the  PBMB.  It  consists  of  a  series  of  timed  exercises  that 
require  the  subject  to  determine  an  aircraft's  position 
relative  to  a  target.  This  test  is  used  in  both  the  pilot 
and  navigator  selection  process  and  assesses  the  applicant's 
ability  to  orient  their  location  from  a  map  view  to  an 
outside  aerial  view  of  the  ground.  This  particular  skill  is 
frequently  used  in  aviation. 

For  this  subtest,  the  computer  screen  is  divided  into 
two  sections.  A  tracker  map  is  displayed  on  the  left  side 
of  the  screen  which  shows  the  location  of  the  Unmanned 
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Aerial  Vehicle  (UAV)  and  the  direction  is  indicated  by  an 
arrow.  North  is  always  oriented  at  the  top  of  the  tracker 
map.  The  image  that  is  displayed  on  the  right  side  of  the 
screen  is  the  target.  The  target  is  shown  as  it  is  viewed 
through  the  UAVs  camera,  depicted  below  in  Figure  3.  This 
camera  is  mounted  beneath  an  UAV  and  will  always  point 
straight  ahead  of  the  UAVs  direction  of  flight. 


Camera  View 


When  you  hear  which  parking  lot  to  image  you  must  move  the  mouse  pointer 
over  that  parking  lot  and  click  the  left  mouse  button.  While  North  is  always  at  the 
top  of  your  Tracker  Map,  this  is  not  always  true  for  the  Camera  View.  You  must 
orient  the  Camera  View  with  the  top  representing  the  direction  the  UAV  is  heading. 


Figure  3.  Instruction  page.  Actual  test  does  not 
display  heading  information  or  an  arrow  but  it 
does  show  the  same  two  camera  views  during  the  DOT 
test . 


The  target  used  in  this  subtest  consists  of  a  building 

surrounded  by  four  parking  lots.  One  parking  lot  is  located 

on  each  cardinal  heading.  The  participant  is  asked  to 

select  one  of  the  surrounding  parking  lots  as  quickly  as 

possible.  The  specific  targeting  instructions  are  received 

verbally  through  headphones  as  well  as  written  below  the 

target  map.  The  participant  must  move  the  mouse  over  the 
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parking  lot  to  be  chosen  and  then  click  the  left  mouse 
button  when  the  arrow  is  displayed  over  that  corresponding 
parking  lot.  The  examinee  will  receive  feedback  on  their 
selection.  A  green  circle  appears  around  the  selected 
parking  lot  when  correctly  identified.  The  elapsed  time 
reguired  to  make  the  decision  is  displayed  inside  the 
circle.  If  an  incorrect  parking  lot  is  selected,  a  red 

circle  will  appear  and  the  associated  elapsed  time  is 
displayed  inside  that  circle. 

The  test  was  designed  to  provide  a  total  of  eight 

practice  guestions  before  the  scored  test  began.  By 

selecting  the  "previous"  button  upon  completion  of  the 

eighth  question,  the  practice  session  could  be  repeated  as 
many  times  as  desired.  There  are  differences  between  the 
actual  test  and  the  practice  test.  During  the  practice 
session,  if  an  incorrect  target  is  selected,  the  correct 
parking  lot  will  be  displayed,  this  is  not  so  during  the 
actual  test.  The  second  difference  is  that  after  a  practice 
question  is  answered,  the  participant  must  select  the 
"continue"  button  to  advance  to  the  next  question,  but  in 
the  actual  test  the  program  advances  automatically.  The 
participants  are  told  that  their  score  for  the  direction 
orientation  task  will  be  calculated  using  a  combination  of 
speed  and  accuracy  of  the  selected  answers. 

2.  Dichotic  Listening  Task  (DLT) 

Pilots  and  flight  officers  often  listen  to  multiple 
radio  frequencies  at  the  same  time  during  a  flight  and  it  is 
imperative  that  they  are  able  to  focus  on  a  particular 
callsign  or  other  important  radio  transmissions. 
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During  the  DLT  portion  of  the  PBMB,  the  participant 
hears  a  series  of  numbers  and  letters  presented  to  each  ear 
through  a  pair  of  headphones,  as  displayed  in  Figure  4.  The 
participant  will  be  verbally  asked  to  monitor  a  "target"  ear 
for  even  or  odd  numbers.  When  an  even  number  is  heard  in 
the  "target"  ear,  the  examinee  is  required  to  press  the 
trigger  of  the  joystick  located  in  their  right  hand.  When  an 
odd  number  is  presented  in  the  "target"  ear,  the  individual 
is  required  to  press  the  thumb  button  of  the  throttle 
located  in  their  left  hand. 


Figure  4.  Example  of  how  the  DLT  test  is  presented  to 
the  subject. 

The  DLT  test  is  one  minute  forty  seconds  in  length  and 
scoring  is  determined  by  how  quickly  and  accurately  the 
participant  responds  to  each  number  presented  in  the 
"target"  ear.  It  is  scored  as  incorrect  if  an  individual 
responds  to  any  number  being  presented  to  the  non-target 
ear . 


3. 

Vertical 

Tracking 

Test  (VTT) 

The 

vertical 

tracking 

test  (VTT) 

takes  one  minute 

to 

complete 

and  measures  an 

applicant' s 

ability 

to  track 

a 

moving  target  on 

a  vertical  axis. 

For  this 

subtest. 

a 

yellow  airplane  and  a  red  crosshair  are  displayed  on  the 
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left  side  of  the  computer  screen.  The  yellow  airplane  moves 
up  and  down  on  the  screen  at  increasingly  faster  speeds. 
The  participant  can  only  control  the  red  crosshair.  The 
object  of  the  test  is  to  keep  the  airplane  inside  the  red 
crosshair  for  as  long  as  possible. 

In  order  to  move  the  crosshair  up  and  down  the  screen, 
the  participant  uses  the  throttle  controls  located  to  their 
left.  When  the  throttle  is  moved  up,  the  crosshair  will 
move  up  on  the  screen  and  conversely,  moving  the  throttle 
down  causes  the  crosshair  to  move  down.  If  the  participant 
is  successful  in  targeting  the  airplane,  the  crosshair  will 
turn  green  and  when  the  targeting  lock  is  lost,  it  returns 
to  red. 

Three  scores  are  recorded  for  this  task.  The  first  is 
a  total  number  of  correct  responses.  A  correct  response  is 
recorded  when  the  pixel  distance  between  the  crosshair  and 
aircraft  is  within  a  programmed  range  at  a  predetermined 
time  check  during  the  task.  The  second  score  recorded  is  an 
error  average  distance.  This  distance  is  the  average 
distance  observed  between  the  center  point  of  the  crosshair 
and  the  center  point  of  the  aircraft  over  the  duration  of 
the  subtest.  The  third  score  is  for  the  total  number  of 
redirects  recorded  during  the  task.  A  redirect  is  recorded 
when  the  subject  maintains  the  crosshair  on  top  of  the 
aircraft  for  brief  amount  of  time  causing  the  aircraft  to 
change  course. 

During  the  VTT,  as  the  participant  correctly  targets 
the  airplane,  it  starts  moving  faster  thus  progressively 
increasing  the  level  of  difficulty.  There  are  three  speeds 
with  which  the  airplane  can  move  and  thus  three  levels  of 
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difficulty.  The  VTT  was  designed  to  measure  how  many 
airplane  redirects  the  participant  can  induce. 

4.  Airplane  Tracking  Test  (ATT) 

The  fourth  subtest  in  the  PBMB  is  the  airplane  tracking 
test.  This  test  takes  one  minute  to  complete  and  measures 
an  individual's  ability  to  track  a  moving  target  in  two 
dimensions.  An  airplane  and  red  targeting  crosshairs  are 
displayed  on  the  screen.  The  basis  for  this  examination  is 
for  the  participant  to  imagine  that  they  are  is  attempting 
to  target  the  airplane  shown  on  the  screen.  The  individual 
attempts  to  keep  the  crosshair  centered  on  the  airplane  as 
it  moves  around  the  screen.  The  crosshairs  will  turn  green 
when  the  airplane  is  successfully  targeted. 

The  examinee  has  no  control  over  the  speed  or  direction 
of  the  target.  The  test  subject  manipulates  the  crosshairs 
by  moving  the  joystick  located  on  their  right.  When  the 
joystick  is  moved  to  the  right  the  crosshairs  move  right. 
Conversely,  when  the  participant  moves  the  joystick  to  the 
left,  the  targeting  crosshairs  move  to  the  left.  When  the 
joystick  is  pushed  forward  the  crosshairs  move  up  and  when 
the  joystick  is  pulled  back  the  crosshairs  move  down. 

The  scores  recorded  for  the  ATT  are  similar  to  the 
three  described  in  the  VTT  section.  As  the  participant 
correctly  targets  the  airplane,  it  starts  moving  faster  thus 
progressively  increasing  the  level  of  difficulty.  The  ATT 
was  also  designed  to  measure  how  many  airplane  redirects  the 
participant  can  induce. 
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5.  Airplane  Tracking  Test  and  Vertical  Tracking  Test 
(ATT/VTT) 

The  ATT/VTT  takes  two  minutes  to  complete  and  is  the 
first  PBMB  subtest  that  combines  two  tasks.  Both  of  the 
tasks  required  in  this  test  have  been  previously  performed 
individually.  During  this  section,  the  subject  is  assessed 
on  their  ability  to  perform  these  tasks  simultaneously. 

The  examinee  is  required  to  use  the  joystick  to  target 
the  airplane  on  the  right  side  of  the  computer  screen  which 
is  moving  in  two  dimensions  and  operate  the  throttle  control 
to  target  the  airplane  moving  along  the  vertical  axis  on  the 
left  side  of  the  screen,  see  Figure  5. 


Figure  5.  Display  used  during  the  ATT/VTT.  The 

aircraft  and  crosshair  shown  on  the  left  are  for 
the  VTT  subset. 
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The  test  subject  must  attend  to  each  task  equally 
during  the  ATT/VTT  because  the  scoring  incorporates  how  well 
both  airplanes  are  accurately  targeted. 

6.  VTT/ATT/DLT  Multi-task  Combination 

This  multi-task  combination  test  takes  three  minutes  to 
complete  and  simulates  the  high  mental  workload  of  some 
aviation  tasks.  This  test  is  a  mixture  of  three  tasks 
previously  performed.  The  VTT,  ATT,  and  DLT  are  all 
administered  during  this  subtest.  The  purpose  of  the  multi¬ 
task  combination  test  is  to  assess  the  subject's  eye-hand 
coordination  while  performing  the  dichotic  listening  test. 

The  joystick  and  throttle  operate  in  the  same  manner  as 
the  VTT/ATT  subtest  while  the  dichotic  listening  task  is 
also  performed. 

The  participant  must  attend  to  all  three  tasks  equally. 
Scoring  for  the  multi-task  combination  subtest  tracks  the 
accuracy  of  targeting  both  airplanes,  the  number  of 
redirects,  and  the  accuracy  and  speed  of  the  responses  to 
the  DLT . 

7 .  VTT/ATT/EP  Scenario 

The  emergency  scenario  test  is  intended  to  replicate 
the  difficult  task  of  dealing  with  problems  while 
maintaining  flight  (a  task  pilots  need  to  be  able  to 
perform) .  While  flying  an  aircraft,  emergencies  are  an  ever¬ 
present  risk.  Making  inaccurate  decisions  and  untimely 
responses  can  be  very  unforgiving.  It  is  also  important 
that  the  pilot  continues  to  fly  the  airplane  while  dealing 
with  an  emergency  scenario. 
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This  two  minute  test  taps  an  individual's  eye-hand 
coordination  plus  memory  capacity.  The  emergency  scenario 
test  is  the  final  subtest  in  the  PMBM.  It  utilizes  the 
vertical  tracking  test  and  the  airplane  tracking  task  while 
the  participant  responds  to  an  emergency  audio  and  visual 
warning.  This  warning  will  identify  one  of  three  emergency 
scenarios  that  are  given  during  the  subtest. 

The  participant  must  carefully  read  the  detailed 
instructions  provided  and  memorize  the  associated  responses 
for  each  emergency  scenario.  The  applicant  will  be  required 
to  respond  to  all  three  emergency  scenarios  over  the  course 
of  this  subtest.  An  example  of  one  emergency  scenario  is  a 
verbal  warning  such  as  "Fire,  Fire,  Fire"  followed  by  the 
illumination  of  a  fire  light.  The  test  subject  is  then 
required  to  respond  to  the  fire  emergency  using  the 
responses  described  in  the  instructions.  Fuel  and  power 
gauges  are  adjusted  to  the  appropriate  settings.  The  gauge 
displays  are  at  the  bottom  of  the  screen,  see  Figure  6. 
Each  emergency  contains  three  actions  which  can  be 
accomplished  by  using  the  throttle  located  to  the  left.  Two 
knobs  and  a  button  are  required  to  manipulate  the 
appropriate  gauges  and  set  them  to  the  correct  position  for 
the  specific  emergency  displayed.  If  an  incorrect  response 
or  no  response  is  given,  the  screen  will  turn  red.  The 
applicant  is  then  notified  visually  that  the  aircraft 
systems  are  operating  under  duress.  If  no  response  is  given 
within  the  predetermined  time,  the  screen  will  reset  in 
preparation  for  the  next  emergency. 
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Figure  6.  The  VTT/ATT/EP  as  viewed  by  the  subject. 

Notice  the  fuel  and  power  gauges  added  to  the 
bottom  right  of  the  screen. 

The  participant  must  continue  to  perform  the  VTT  and 
the  ATT  tasks  while  dealing  with  the  emergency  scenario. 
All  three  tasks  are  considered  in  the  final  score  for  this 
subtest . 
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III.  RESULTS 


Of  the  40  volunteers,  four  were  female  and  36  male,  the 
ages  of  the  subjects  ranged  from  27  to  43  years  old  and  the 
median  and  mean  ages  were  both  33  years. 

For  the  purpose  of  this  study,  an  individual  with 
flight  experience  was  defined  as  those  participants  who 
received  any  formal  pilot  or  flight  officer  training,  20  met 
this  general  criterion.  This  group  was  comprised  of  two 
Naval  Flight  Officers  (NFO)  and  18  pilots  qualified  to  fly 
jets,  propeller  aircraft  or  helicopters.  The  flight 
experience  group  also  contained  one  participant  with  only  a 
private  pilot's  license  (75  hours  of  flight  time),  a 
participant  who  started  but  did  not  complete  flight  school 
(200  hours  flight  time) ,  and  one  participant  that 
transitioned  from  a  NFO  to  a  pilot.  The  mean  flight  time 
was  1423.75  hours  with  the  minimum  number  of  hours  being  75 
and  3000  being  the  maximum.  One  participant  did  have  flight 
time  as  an  observer  but  this  qualification  did  not  meet  this 
study' s  definition  of  flight  experience  and  therefore  they 
were  not  included  in  the  flight  experience  group. 

Only  two  of  the  participants  with  aviation  experience 
held  a  current  instrument,  instructor  or  Naval  Air  Training 
and  Operating  Procedures  Standardization  (NATOPS) 
qualification  at  the  time  of  the  experiment.  The  mean  time 
since  the  last  flight  was  29  months.  The  longest  time 
observed  was  120  months  and  the  shortest  was  seven  months 
since  their  last  flight. 

Participants  were  asked  how  often  they  played  video 

games  and  if  any  of  those  games  were  flight  simulators. 
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Fifteen  volunteers  reported  playing  video  games  on  a  monthly 
basis  and  only  two  indicated  they  occasionally  played  flight 
simulator  games.  The  mean  time  spent  playing  video  games 
per  month  was  5.35  hours  with  the  highest  being  60  hours. 

All  exploratory  analysis  conducted  in  this  thesis  was 
accomplished  using  JMP  7  and  SPSS  personal  computer  software 
packages.  Unless  otherwise  noted  all  results  are  quoted  at 
the  two-tailed  0.05  significance  level. 

A.  SPATIAL  ORIENTATION  SKILLS 

1.  Direction  Orientation  Test  (DOT) 

A  total  of  48  guestions  were  asked  in  the  DOT  subtest 
and  each  participant  received  two  scores.  One  score  was  for 
the  total  number  of  correct  responses  and  a  second  score  for 
the  total  response  time  for  all  correct  and  incorrect 
answers . 

The  mean  score  of  correct  responses  for  the  aviation 
experience  group  was  39.95  with  a  standard  deviation  of  5.09 
while  the  no  experience  group  had  a  mean  of  38.15  and  a 
standard  deviation  of  6.52.  These  two  means  were  not 
statistically  significant  (F  =  0.95,  p  =  .34) . 

The  mean  total  response  time  observed  for  the  group 
with  aviation  experience  was  two  minutes  twenty  three 
seconds  with  a  standard  deviation  of  one  minute  thirteen 
seconds.  The  group  with  no  experience  took  slightly  longer 
with  a  mean  time  of  two  minutes  thirty  seconds  and  a 
standard  deviation  of  one  minute  nineteen  seconds,  these 
means  were  not  statistically  significant  (F  =  0.04,  p  =  .84) 


A  composite  score  was  created  to  address  the  speed  and 
accuracy  trade-off  of  the  DOT.  The  total  number  correct  and 
total  response  time  for  each  individual  score  were 
standardized  and  combined  (weighted  equally) .  The  difference 
between  the  mean  composite  scores  of  the  two  groups  (0.12 
for  those  with  aviation  experience  and  -0.12  for  those 
without)  was  not  statistically  significant  ( F  =  0.57,  p  = 
.46)  . 

B.  LISTENING  SKILLS 

1.  Dichotic  Listening  Test  (DLT) 

A  total  of  16  questions  were  asked  during  the  one 
minute  forty  second  DLT  portion  of  the  psychomotor  test. 
Those  with  aviation  experience  had  a  mean  number  of  correct 
responses  of  11.9  and  a  standard  deviation  of  4.52  while 
those  with  no  aviation  experience  had  a  mean  number  of 
correct  responses  of  10.15  and  a  standard  deviation  of  4.34 
(see  Table  1) .  The  means  for  the  number  of  correct 
responses  in  the  DLT  were  not  statistically  significant  (F  = 
1.56,  p  =  .22)  . 

The  mean  total  response  time  for  correct  responses 
observed  for  the  group  with  aviation  experience  was  0:16.11 
seconds  with  a  standard  deviation  of  0:06.12  seconds.  The 
group  with  no  experience  had  a  mean  response  time  of  0:15.94 
seconds  and  a  standard  deviation  of  0:07.83  seconds.  An 
average  time  for  a  correct  response  was  calculated  by 
dividing  the  total  correct  time  by  the  total  number  of 
correct  responses.  The  number  correct  and  average  time  were 
then  standardized  and  weighted  equally  to  calculate  a 

composite  score.  The  difference  between  the  mean  composite 
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scores  of  the  two  groups  (0.19  for  those  with  aviation 
experience  and  -0.19  for  those  without)  was  not 
statistically  significant  (F  =  3.178,  p  =  .083) . 


DLT  Subtest 

EXPERIENCE 

NO  EXPERIENCE 

Number  Correct 

Adjusted  Number 
Correct 

Total  Response  Time 
for  Correct 
Responses 

Mean  STD  DEV 

Mean  STD  DEV 

Mean  STD  DEV 

11.9  4.52 

10.15  4.34 

20.83  7.906 

17.76  7.602 

00:16.1  00:06.1 

00:15.9  00:07.8 

Table  1.  Summary  of  both  adjusted,  unadjusted  response 
times  for  the  DLT  subtest. 


C.  EYE-HAND  COORDINATED  TRACKING  SKILLS 

1.  VTT ,  ATT,  and  VTT/ATT  Tests 

For  the  purpose  of  this  study,  we  defined  eye-hand 
coordinated  tracking  skills  as  the  one  and  two  dimensional 
tracking  accomplished  with  the  joystick  and  throttle.  The 
subtests  containing  tracking  skills  varied  in  duration.  The 
shortest  time  was  one  minute  for  the  VTT  and  ATT  with  the 
longest  being  three  minutes  for  the  VTT/ATT/DLT. 

For  each  of  the  VTT  and  ATT  tracking  skills,  a  total  of 
three  scores  were  recorded.  The  first  score  was  the  number 
of  correct  responses,  the  second  was  the  average  distance 
error  between  the  crosshairs  and  the  airplane,  and  the  third 
score  was  the  total  number  of  times  the  participant  caused 
the  aircraft  to  redirect  because  of  capture.  It  should  be 
noted  that  the  lower  the  error  average  distance  the  closer 
the  crosshairs  remained  to  the  aircraft  during  the  test. 
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The  correlations  between  the  different  tracking  tasks 
and  the  different  tracking  measures  were  all  highly 
correlated  (see  Table  2)  .  The  first  three  rows  in  each 
matrix  are  the  number  correct,  error  average,  and  number  of 
redirects  for  the  VTT .  Rows  three  through  six  are  the 
number  correct,  error  average,  and  number  of  redirects  for 
the  ATT  test.  Row  seven  is  a  sum  of  the  number  correct  for 
the  VTT  and  ATT  test.  Row  eight  is  an  average  of  the  error 
average  for  the  VTT  and  ATT  test  and  row  nine  is  the  sum  of 
the  total  number  of  redirects  for  the  VTT  and  ATT  test  in 
each  subtest.  Since  the  number  correct,  error  average,  and 


number 

of 

redirects  are 

so  highly  correlated. 

only 

the 

number 

of 

redirects  will 

be  reported  from 

this 

point 

on 

(number 

of 

redirects  was 

selected  because 

the 

PBMB 

was 

designed  to  use  it  as  its  primary  measure  of  tracking 
skill)  . 
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VTT/ATT/DLT  Subtest 

VTT 

Correct 

VTT 

Error 

Avg 

VTT 

Redirect 

ATT 

Correct 

ATT 

Error 

Avg 

ATT 

Redirect 

Total 

Correct 

Total 

Error 

Avg 

Total 

Redirect 

VTT  Correct 

1.00 

VTT  Error  Avg 

-0.93 

1.00 

VTT  Redirect 

0.99 

-0.92 

1.00 

ATT  Correct 

0.71 

-0.62 

0.70 

1.00 

ATT  Error  Avg 

-0.60 

0.61 

-0.59 

-0.89 

1.00 

ATT  Redirect 

0.70 

-0.62 

0.69 

0.99 

-0.90 

1.00 

Total  correct 

0.93 

-0.85 

0.92 

0.92 

-0.80 

0.91 

1.00 

Total  Error  Avg 

-0.84 

0.88 

-0.83 

-0.85 

0.91 

-0.86 

-0.92 

1.00 

Total  Redirect 

0.91 

-0.83 

0.91 

0.93 

-0.82 

0.93 

1.00 

-0.92 

1.00 

VTT/ATT/EP  Subtest 

VTT 

ATT 

Total 

VTT 

Error 

VTT 

ATT 

Error 

ATT 

Total 

Error 

Total 

Correct 

Avg 

Redirect 

Correct 

Avg 

Redirect 

Correct 

Avg 

Redirect 

VTT  Correct 

1.00 

VTT  Error  Avg 

-0.94 

1.00 

VTT  Redirect 

0.99 

-0.95 

1.00 

ATT  Correct 

0.68 

-0.71 

0.67 

1.00 

ATT  Error  Avg 

-0.70 

0.69 

-0.67 

-0.85 

1.00 

ATT  Redirect 

0.68 

-0.71 

0.68 

0.99 

-0.87 

1.00 

Total  Correct 

0.94 

-0.92 

0.94 

0.88 

-0.83 

0.88 

1.00 

Total  Error  Avg 

-0.86 

0.89 

-0.86 

-0.86 

0.95 

-0.87 

-0.94 

1.00 

Total  Redirect 

0.92 

-0.92 

0.92 

0.90 

-0.84 

0.91 

0.99 

-0.94 

1.00 

Table  2 .  Correlation  matrix  of  scores  for  the 
VTT/ATT/DLT  and  VTT/ATT/EP . 

2 .  Vertical  Tracking  Test  (VTT) 

The  experienced  group  induced  redirects  at  a 
significantly  higher  rate  than  those  without  aviation 

experience,  17.9  and  15.5,  respectively,  F(l,39)  =  4.59,  p  = 
. 04  (see  Table  3 ) . 

3.  Airplane  Tracking  Test  (ATT) 

Again,  the  experienced  group  induced  redirects  at  a 

significantly  higher  rate  than  those  without  aviation 

experience,  9.95  and  6.05,  respectively,  F(l,39)  =  7.92,  p  = 
. 008  (see  Table  3) . 
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4. 


Vertical  Tracking  Test/Airplane  Tracking  Test 
(VTT/ATT) 


The  total  number  of  redirects  for  the  VTT/ATT  subtest 
was  calculated  by  adding  the  VTT  number  of  redirects  with 
the  ATT  number  of  redirects.  The  experienced  group  induced 
redirects  at  a  significantly  higher  rate  than  those  without 
aviation  experience,  26.55  and  18.5,  respectively,  F(l,39)  = 
9.72,  p  =  .004  (see  Table  3) . 

5.  VTT/ATT/DLT  Multi-task  Combination 

In  this  subtest,  the  VTT/ATT  scores  were  combined  as  in 
the  previous  section.  The  number  of  correct  responses  from 
the  DLT  was  also  recorded. 

The  experienced  group  induced  redirects  during  the 
VTT/ATT/DLT  subtest  at  a  significantly  higher  rate  than 
those  without  aviation  experience,  37.35  and  23.15, 
respectively,  F(l,39)  =  10.66,  p  =  .002.  However,  the  DLT 
mean  number  correct  was  not  statistically  different  between 
the  group  with  aviation  experience  and  those  without 
aviation  experience,  21.1  correct  and  20.65  correct 
respectively,  F(l,39)  =  0.06,  p  =  .81  (see  Table  3) .  When  a 
composition  score  for  speed  and  accuracy  was  calculated 
using  the  average  time  for  a  correct  response  (same  method 
described  in  the  DLT  section) ,  the  DLT  mean  composite  scores 
between  the  two  groups  were  not  significant  (F  =  0.11,  p  = 
.75) 

To  determine  the  impact  of  the  DLT  on  tracking  skills, 
the  mean  number  of  redirects  was  adjusted  for  the  one  minute 
time  difference  between  the  VTT/ATT  subtest  and  the 
VTT/ATT/DLT  subtest  so  these  two  tests  could  be  directly 
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compared  (see  Table  3)  .  Both  groups  experienced  a  drop  in 
the  mean  number  of  redirects  when  the  additional  DLT  task 
was  added.  The  adjusted  mean  number  of  redirects  for  the 
VTT/ATT  was  compared  to  the  mean  number  of  redirects  for  the 
VTT/ATT/DLT  for  each  group.  Assuming  unequal  variance,  the 
means  for  the  group  with  aviation  experience  were  39.83  and 
37.35  (Prob.  >  |t|  0.59)  and  the  means  for  the  group  without 
aviation  experience  were  27.75  and  23.15  (Prob.  >  |t|  0.25). 
Although  these  means  are  not  statistically  different,  they 
do  trend  in  the  expected  direction. 

The  adjusted  DLT  mean  number  of  correct  responses  (see 
Table  1)  was  compared  to  the  mean  number  of  correct 
responses  for  the  DLT  section  of  the  VTT/ATT/DLT  for  each 
group.  Assuming  unequal  variance,  the  means  for  the  group 
with  aviation  experience  were  20.83  and  21.1  (Prob.  >  |t| 
0.89)  and  the  means  for  the  group  without  aviation 
experience  were  17.76  and  20.65  (Prob.  >  |t|  0.22)  .  These 
means  are  also  not  statistically  different,  however  it  is 
interesting  to  note  that  both  groups  performed  better  on  the 
DLT  the  second  time  they  took  it  (VTT/ATT/DLT)  even  though 
they  were  also  performing  a  dual-tracking  task. 

6.  VTT/ATT/EP  Scenario 

The  emergency  scenario  subtest  was  two  minutes  long  and 
this  section  was  treated  similar  to  the  VTT/ATT/DLT.  The 
VTT/ATT  number  of  redirects  for  the  VTT/ATT/EP  subtest  was 
combined  in  a  single  score.  The  number  of  correct  emergency 
responses  was  tallied  with  a  maximum  score  achievable  being 
three.  The  elapsed  time  to  respond  to  each  emergency  was 
also  recorded. 
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The  experienced  group  induced  redirects  during  the 
VTT/ATT/EP  subtest  at  a  significantly  higher  rate  than  those 
without  aviation  experience,  21.05  and  11.9,  respectively, 
F(l,39)  =  12.49,  p  =  .001  (see  Table  3) .  However,  the  mean 
number  of  correct  EP' s  was  2.2  for  the  group  with  experience 
and  1.85  without  aviation  experience,  these  means  were  not 
statistically  different  ( F  =  1.12,  p  =  .30) .  When  a 

composite  score  taking  into  account  the  speed  and  accuracy 
of  responses  during  the  emergency  scenario  was  calculated 
(using  the  same  method  described  in  the  DLT  section)  ,  the 
composite  mean  scores  (0.40  for  the  group  with  experience 
and  -0.40  for  those  without  experience)  between  the  two 
groups  were  not  statistically  significant  (F  =  2.05,  p  = 
.16)  . 

Since  both  the  VTT/ATT  and  VTT/ATT/EP  subtests  were  two 
minutes  long,  no  adjustment  to  the  mean  number  of  redirects 
was  necessary  to  determine  the  impact  of  tracking  while 
performing  the  EP  task  (see  Table  3)  .  Both  groups 
experienced  a  drop  in  the  mean  number  of  redirects. 


Assuming  unequal 

variance , 

the 

means  for 

the 

number  of 

redirects  of  the 

aviation  experience  group 

were 

39.83  and 

31.58  (Prob.  > 

1 1 1  0.049) 

and 

the  means 

for 

the  group 

without  aviation 

experience 

were 

27.75  and 

17 . 85 

(Prob.  > 

| t |  0.011).  This  mean  difference  is  statistically  different 

and  trended  in  the  expected  direction  for  both  groups. 
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VTT  Subtest 

EXPERIENCE 

NO  EXPERIENCE 

Number  Redirects 

Adjusted  Number 
Redirects 

Mean  STD  DEV 

Mean  STD  DEV 

17.9  3.06 

15.5  3.97 

53.7  9.18 

46.5  11.9 

ATT  Subtest 

EXPERIENCE 

NO  EXPERIENCE 

Number  Redirects 

Adjusted  Number 
Redirects 

Mean  STD  DEV 

Mean  STD  DEV 

9.95  4.37 

6.05  4.39 

14.93  6.56 

9.6  7.11 

ATT/VTT  Subtest 

EXPERIENCE 

NO  EXPERIENCE 

Number  Redirects 

Adjusted  Number 
Redirects 

Mean  STD  DEV 

Mean  STD  DEV 

26.55  8.16 

18.5  8.17 

39.83  12.24 

27.75  12.26 

ATT/VTT/DLT  Subtest 

EXPERIENCE 

NO  EXPERIENCE 

Number  Redirects 

Adjusted  Number 
Redirects 

#  DLT  Correct 

DLT  Average  Time 

Mean  STD  DEV 

Mean  STD  DEV 

Mean  STD  DEV 

Mean  STD  DEV 

37.35  16.26 

23.15  10.67 

37.35  16.26 

23.15  10.67 

21.1  4.54 

20.65  7.01 

1.34  0.49 

1.53  0.61 

EXPERIENCE 

NO  EXPERIENCE 

ATT/VTT/EP  Subtest 

Number  Redirects 

Adjusted  Number 
Redirects 

#  EP  Correct 

EP  Average  Time 

Mean  STD  DEV 

Mean  STD  DEV 

Mean  STD  DEV 

Mean  STD  DEV 

21.05  8.95 

11.9  7.34 

31.58  13.42 

17.85  11.02 

2.2  0.77 

1.85  1.27 

28.4  4.5 

23.9  12.8 

Table  3 . 


Summary  of 
scores  for 
EP  for  the 


both  the  unadjusted  and  adjusted 
the  number  of  redirects,  DLT  and 
tracking  skill  subtests. 


D.  PARTICIPANT  DEMOGRAPHICS 

Prior  to  taking  the  PBMB,  participants  were  asked  a 
series  of  questions  about  their  perceived  eye-hand 
coordination  skills  and  multitasking  abilities.  These 
responses  along  with  the  participants '  age,  gender,  aviation 
experience,  amount  of  video  games  played,  ability  to  juggle, 
and  ability  to  dribble  a  basket  ball  were  analyzed  using  a 
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stepwise  regression  to  predict  the  total  number  of  redirects 
for  the  VTT/ATT/EP  subtest.  Only  aviation  experience  was 
significant  (p  =  .001) .  When  the  same  predictors  were  used 
to  model  the  number  of  redirects  for  the  VTT/ATT/DLT 
subtest,  aviation  experience  (p  =  .004)  and  gender  (p  = 
.047)  were  statistically  significant  predictors.  Note,  there 
were  only  four  females  in  the  study  and  only  one  had  flight 
experience  -  this  small  sample  size  and  unequal  cells 
preclude  any  predictions  based  on  gender. 
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IV.  DISCUSSION 


Certain  subsets  of  the  psychomotor  test  did  support  our 
hypothesis  that  individuals  having  flight  experience  would 
on  average,  score  higher  than  those  without  flight 
experience . 

The  most  significant  results  were  the  mean  scores  for 
the  eye-hand  coordinated  tracking  tasks.  These  skills 
included  tracking  two  airplanes  with  a  throttle  and 
joystick.  While  we  focused  on  the  total  number  of 
redirects,  either  the  total  number  correct  responses  or  the 
average  error  distance  would  have  detected  the  difference  in 
tracking  ability  between  the  experienced  and  non-experienced 
groups.  It  has  been  claimed  that  eye-hand  coordination  and 
tracking  skills  are  important  in  the  field  of  aviation 
(Mangos,  2005)  and  these  findings  suggest  that  our  selected 
group  of  aviator  not  only  had  these  skills  but  that  the  PBMB 
was  able  to  detect  them. 

The  VTT ,  ATT,  VTT/ATT,  VTT/ATT/DLT,  and  the  VTT/ATT/EP 
sections  all  produced  significant  results  when  comparing  the 
mean  number  of  redirects  between  the  two  groups. 

The  aviation  experience  group  and  the  group  without 
aviation  experience  showed  the  largest  significant 
difference  when  performing  the  VTT/ATT/EP  (F  =  12.49,  p  = 
.001) .  When  the  adjusted  mean  number  of  redirects  for 
VTT/ATT/  subtask  were  compared  to  the  adjusted  mean  number 
of  redirects  for  the  VTT/ATT/EP  subtask,  both  groups  had  a 
significant  decrease  in  mean  scores.  This  drop  in 
performance  was  expected  because  participants  were  reguired 
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to  manipulate  thumb  wheels  located  on  the  throttle  control 
during  the  emergency  scenario  which  diverted  attention  away 
from  aircraft  tracking. 

The  VTT/ATT/DLT  subtest  (although  not  as  robust  as  the 
VTT/ATT/EP)  resulted  in  a  significant  difference  between  the 
aviators  and  non-aviators  (F  =  10.66,  p  =  .002)  and  was 
consistent  with  the  findings  of  Delaney  (1992) .  The 
difference  between  the  adjusted  mean  redirects  for  the 
VTT/ATT  subtest  and  the  mean  redirects  for  the  VTT/ATT/DLT 
subtest  showed  a  decline  in  performance.  Again,  this  trend 
was  expected  due  to  participants  performing  another  test. 

When  the  adjusted  means  for  the  one  minute  DLT  were 
compared  to  the  DLT  portion  of  the  VTT/ATT/DLT  subtask,  we 
anticipated  that  the  mean  of  the  average  time  per  correct 
response  for  both  groups  would  be  higher  when  the 
participant's  attention  was  divided  among  tasks.  However, 
the  results  showed  that  both  groups  mean  average  time  for  a 
correct  response  decreased.  We  observed  a  marginal 
improvement  in  the  experienced  group' s  performance  from  a 
mean  of  1.34  to  1.30  seconds  and  a  more  substantial 
improvement  for  the  group  with  no  experience  from  1.53  to 
1.26  seconds.  Although  the  observed  decrease  for  the  two 
groups  was  not  statistically  different,  this  improvement  was 
unexpected.  The  most  logical  explanation  for  this 
improvement  can  be  attributed  to  the  novelty  of  the  DLT  task 
causing  participants  trouble.  However,  by  the  time  they  took 
the  VTT/ATT/DLT  section,  they  may  have  figured  out  the 
procedure  and  thus  their  scores  improved.  This  explanation 
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is  even  more  likely  given  the  participants  comments  about 
misunderstanding  during  the  practice  portion  of  the  DLT 

subtest  (see  the  Observations  and  Recommendations  section 
below)  . 

The  DOT  did  not  distinguish  between  the  experienced 
aviators  and  the  non-experienced  group.  Since  the  Navy 
currently  uses  a  paper  and  pencil  style  spatial  abilities 
test  in  the  ASTB  which  is  different  from  the  DOT  spatial 
ability  test,  they  may  have  to  continue  relying  on  that 
test.  The  instructions  for  the  DOT  test  state  that  the 

scoring  will  be  determined  by  the  speed  and  accuracy  of  a 
response.  Two  scores  were  provided  in  the  data  output  for 
this  section.  One  score  was  a  total  accumulated  time  for 

correct  and  incorrect  responses  and  the  other  reported  the 
total  number  of  correct  responses.  Finding  the  ideal 
weighting  of  these  scores  in  a  composite  score  is  beyond  the 
scope  of  this  paper,  however  we  did  look  at  a  composite 
score  based  on  an  equally  weighted  standardize  version  of 
the  number  correct  and  total  response  time  and  found  no 

significant  difference  between  the  mean  scores  for  the  two 
groups.  The  PBMB  DOT  test  was  unable  to  detect  a  difference 
in  spatial  abilities  between  those  with  flight  experience 
and  those  without.  Using  a  more  sophisticated  weighting 
scheme  may  prove  more  fruitful. 

The  DLT  was  another  subtest  of  the  PBMB  that  did  not 
distinguish  between  the  group  with  aviation  experience  and 
the  group  with  no  aviation  experience.  When  the  mean  number 
correct  responses  for  the  DLT  was  examined,  there  was  no 
difference  between  groups.  Although  the  composite  score  of 
speed  and  accuracy  did  not  differ  between  groups  there  was 
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an  increase  in  the  F  statistic,  which  was  F  =  1.56  for  the 
mean  number  correct  and  F  =  3.18  for  the  composite  score. 
The  composite  score  was  based  on  an  equally  weighted 
standardize  version  of  the  number  correct  and  total  time  for 
correct  responses.  A  more  sophisticated  model  of  weighting 
the  number  correct  and  total  time  for  correct  responses 
might  prove  a  more  robust  measure. 

It  is  apparent  that  those  with  aviation  experience  did 
perform  better  than  those  without  on  the  eye-hand  tracking 
skills  contained  in  the  psychomotor  test.  Either  of  the 
last  two  subtests  can  adequately  measure  this  difference. 
The  DOT  subtest  did  not  support  our  hypothesis,  but  it 
appears  that  dichotic  listening  may  have  affected 
performance  during  the  VTT/ATT/DLT  subtest.  These  two 
sections  may  benefit  from  improvements  in  data  collection, 
and  scoring,  and  further  evaluation  of  how  the  final  scoring 
model  is  produced. 

A.  OBSERVATIONS  AND  RECOMMENDATIONS 

The  directions  included  before  each  subtest  were  found 
to  have  sufficient  detail.  One  interesting  observation  to 
note  pertained  to  the  emergency  procedure  section. 
Participants  were  told  to  adjust  fuel  and  power  levels  to 
either  high,  low,  or  neutral  depending  on  the  emergency  they 
were  presented.  However,  the  gauges  were  displayed  with  a 
red  stripe  located  at  the  top,  green  in  the  middle,  and 
yellow  at  the  bottom,  as  shown  in  Figure  6.  Although  one 
would  normally  assume  that  the  top  of  gauge  is  a  high 
setting,  it  had  a  red  indication  which  may  be  interpreted  as 
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being  low.  This  subtle  difference  in  directions  has  the 
potential  to  cause  some  confusion  and  should  be  clarified  in 
future  test  versions. 

All  practice  secessions  preceding  each  subtest  could 
only  be  performed  once  except  for  the  direction  orientation 
test.  A  few  of  the  volunteers  were  observed  repeating  the 
DOT  practice  session  multiple  times.  This  was  made  possible 
by  selecting  the  "previous"  button  upon  completion  of  the 
eighth  practice  question.  It  is  unknown  whether  this  extra 
practice  had  any  influence  on  the  results  for  the  DOT 
section.  The  directions  should  clearly  state  that 
participants  can  repeat  the  practice  section  of  the  DOT  or 
the  possibility  for  repeated  practice  should  be  eliminated. 

The  practice  session  for  the  DLT  is  different  than  the 
full  DLT  test.  The  30  second  practice  session  does  not 
switch  "target"  ears  while  the  actual  test  does  require  the 
participant  to  shift  attention  from  one  ear  to  the  other 
ear.  This  inconsistency  may  have  caused  some  confusion.  An 
individual  may  not  have  been  expecting  a  switch  in  "target" 
ears  on  the  DLT,  but  by  the  end  of  the  test  they  usually 
understood  the  task.  This  initial  shock  of  switching  ears 
may  explain  why  both  groups  increase  their  DLT  score  later 
during  the  VTT/ATT/DLT  subtest.  Consideration  should  be 
given  to  changing  the  DLT  practice  session  to  include  a 
switch  in  "target"  ears.  Furthermore,  increasing  the  length 
of  the  actual  DLT  may  ensure  sufficient  time  for  learning 
this  novel  task. 

Participants  were  given  an  opportunity  to  provide 
feedback  on  the  psychomotor  test  when  the  questionnaire  was 
completed.  One  issue  mentioned  was  that  the  factory 
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sensitivity  settings  on  the  joystick  and  throttle  were 
awkward.  The  throttle  had  to  be  moved  forward  or  backward 
approximately  2/3  of  the  way  to  get  the  crosshairs  to  begin 
to  move.  Also,  full  right  or  left  movement  of  the  joystick 
was  difficult  to  achieve  without  the  control  unintentionally 
moving  on  the  table  top.  Some  refinement  to  the  sensitivity 
setting  may  be  desirable. 

Another  comment  from  the  majority  of  the  volunteers  was 
the  overall  difficulty  of  the  test.  Of  the  40  participants, 
38  of  them  felt  that  the  test  was  challenging  while  24  found 
it  to  be  extremely  challenging.  Only  two  volunteers 
considered  it  to  be  about  the  right  level  of  difficulty. 
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V.  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 


The  scope  of  this  study  was  limited  and  we  only  touched 
the  surface  of  the  required  analysis  needed  prior  to 
fielding  the  Navy' s  new  psychomotor  test  for  aviator 
selection.  There  exists  a  multitude  of  opportunities  for 
further  research  and  subtle  refinements  in  the  current 
version  of  the  PBMB . 

The  first  area  that  should  be  researched  further 
involves  the  development  and  analysis  of  a  model  to 
correctly  weight  response  time  with  correct  responses  for 
both  DLT  and  DOT  tests.  In  this  study  we  elected  to  equally 
weight  these  scores  which  may  not  be  ideal.  Exploring  a 
better  weighting  scheme  should  be  addressed. 

We  elected  to  only  look  at  subjects  who  had  already 
completed  some  type  of  formal  flight  training.  Valuable 
insight  was  gained  into  the  tests  ability  to  recognize  eye- 
hand  coordination  and  tracking  skills  but  it  is  still 
unknown  whether  or  not  this  was  a  learned  skill  or  an 
inherent  ability.  Furthermore,  the  aviator's  adage  of 
aviate,  navigate,  and  communicate  may  have  played  a  role 
their  disparate  scores  in  the  multi-tracking  task  and 
emergency  scenario  subtests. 

None  of  our  test  subjects  were  under  the  age  of  27  or 

currently  selected  for  aviation  and  awaiting  formal  flight 

training.  Further  research  should  involve  testing  this 

target  population.  These  individuals  should  be  administered 

the  psychomotor  test  prior  to  having  any  aviation 

experience.  They  should  be  tracked  throughout  their  formal 

flight  training  and  upon  completion  or  discontinuation  from 
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flight  school.  Their  flight  school  performance  could  be 
used  to  design  an  appropriate  weighted  model  for  the  PBMB . 
Results  from  a  study  of  this  nature  should  provide  further 
insight  into  the  PBMBs  predictive  capability  and  eventually 
allow  for  a  standard  measure  to  be  established  for 
acceptance  into  flight  school. 
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APPENDIX 


Questionnaire 


Section  1 

1.  Participant  Number: 


2 .  Age : 


3.  Sex: 

a.  Male 

b.  Female 

4.  On  a  scale  of  1  to  5  with  5  being  the  highest, 
would  you  rate  your  overall  hand  eye  coordination? 

5  -  Excellent 

4  -  Above  average 

3  -  Average 

2  -  Below  Average 

1  -  Poor 

5.  Do  you  have  any  aviation  experience? 

a .  Yes 

b .  No 


If 

6. 


you  answered  no  to  question  5  skip 
Is  your  flight  experience  as  a: 

a.  Pilot 

b.  Flight  officer 

c.  Flight  engineer 

d.  Crewmember 


to  question 


13. 


7.  Is  your  aviation  time 

a.  Military 

b.  Civilian 

c.  Both 


8. 


What  type  of  airframe  (circle  all  that  apply) ? 

a.  Fixed  wing  -  Jet 

b.  Fixed  wing  -  propeller 

c.  Helicopter 

d.  Other 


how 
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9.  Approximately  how  many  total  hours  would  you  say  you 
have  (rounding  is  fine) ? 


10.  What  is  the  highest  qualification  you  have  held? 
(Instructor,  mission  commander,  test  pilot  etc)  : 


11.  Are  any  of  your  qualifications  still  current? 

a.  Yes  -  all 

b.  Yes  -  some 
b .  No 


12.  How  long  has  it  been  since  your  last  flight  (in 
months) ? 


13.  On  average,  how  much  time  per  month  do  you  play  video 
games  (in  hours) ? 


14.  Are  any  of  the  video  games  flight  simulators? 

a .  Yes 

b .  No 

15.  On  a  scale  of  1  to  5  with  5  being  the  highest,  circle 
how  athletic  you  think  you  are 

5  -  Should  have  turned  pro 

4  -  Above  average 

3  -  Average 

2  -  Below  average 

1  -  Always  the  last  kid  picked 

16.  Are  you  currently  playing  any  sport  (recreational  or 
organized) ? 

a .  Yes 

b .  No 


ever  played  any  organized  sport? 


17.  Have  you 

a .  Yes 

b .  No 
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18.  What  sport  and  what  position  (example:  baseball,  short 
stop)  ? 


19.  On  a  scale  of  1  to  5  with  5  being  the  highest,  circle 
how  well  would  you  think  you  are  at  multitasking? 

5  -  Excellent 

4  -  Above  average 

3  -  Average 

2  -  Below  Average 

1  -  Poor 

20.  The  game  is  on  the  line  and  it  all  comes  down  to  one 
final  play.  If  the  play  is  made,  your  team  wins,  if  not, 
you  lose.  Do  you  want  to  be  the  one  who: 

a.  Is  in  position  to  make  the  play.  You  have  the 
utmost  confidence  in  your  ability. 

b.  Is  in  the  play  but  the  entire  thing  does  not  hinge 
on  your  actions. 

c.  Is  out  there  to  back  up  the  individual  up  who 
makes  the  game  winning  play. 

d.  Is  not  currently  in  the  game  but  has  a  front  row 

seat . 

e.  Not  applicable  -  wouldn't  be  playing  in  a  sporting 

event 

21.  Can  you  juggle? 

a .  Yes 

b .  No 

22.  How  well  would  you  say  you  can  dribble  a  basketball? 

5  -  Excellent 

4  -  Above  average 

3  -  Average 

2  -  Below  Average 

1  -  Poor 
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Section  2 


1.  How  well  do  you  think  you  did  on  the  test? 
5  -  Excellent 

4  -  Above  average 

3  -  Average 

2  -  Below  Average 

1  -  Poor 


2. 


How  would  you  rate  the  difficulty  of  the  exam? 
5  -  Extremely  challenging 

4  -  Challenging 

3  -  Just  about  right 

2  -  Walk  in  the  park 

1  -  I  could  do  it  with  my  eyes  closed 


3.  What  did  you  like  about  the  exam? 


4.  What  did  you  not  like  about  the  exam? 


Would  you  say  this  is  an  accurate  assessment  of  your  hand- 
eye  coordination  (explain  why  or  why  not) ? 
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